A typical mechanism for implementing user interaction with a computer may involve the use of a graphical user interface (GUI) associated with a computer program running on the computer. Generally speaking, user interface elements (hereafter, UI elements) are aspects of a computer system or program which may be seen (or heard or otherwise perceived or interacted with) by an end user and used to control the operation of a computer system or program through commands and other mechanisms. A GUI emphasizes the use of graphical UI elements which are responsive to user input (e.g., mouse events, keyboard data entry etc.) for interacting with a computer system or program. Common UI elements familiar to most users include buttons (e.g., “OK” or “Cancel”), edit boxes, list boxes, combo boxes, scroll bars, pick lists, pushbuttons, radio buttons components of World Wide Web pages (e.g., hyper-links, documents and images) and the like.
In addition to users directly interacting with UI elements of a program, other computer programs may need to the same. Thus, one program may import, use, search for or otherwise interact with the UI elements of another program. For instance, a test program may use programmatic access to UI elements of another program that it is testing. In such cases, the test program's script may find a UI element in a target program and repeatedly exercise the element to discover any errors.
Some users with special needs (e.g., vision impaired, hearing impaired etc.) may not be able to interact with UI elements of a program in the same manner as those without such special needs. For instance, a visually impaired user may not be able to see the elements of a list box to make a choice from among its list items. There are programs that are specially adapted to assist such users. Such assistive technology programs may use programmatic access to the list box UI element to determine its individual list items and then use a special narrator utility to enable the user to vocally interact with the program. Magnifiers, screen readers and tactile mice are but some examples of such assistive technology.
There are numerous other reasons for one program to interact with UI elements of another program. The interactions may include one program attempting to find a particular UI element of interest in another program it is testing as may be the case in a test program scenario. Also, as in the assistive technology scenario, one program may also want knowledge of the structure or relationship information about various elements of another program's GUI. Besides a structure, one program may also want knowledge of properties (e.g., visual, functional etc.) of the various UI elements of another program.
Regardless of the type of interaction, for one program to interact with the UI elements of another, a programmatic representation of such elements may be needed. FIG. 1 comprises a hierarchy 100 that graphically illustrates a programmatic representation of a typical GUI. More particularly, FIG. 1 illustrates how elements of a GUI can be shown as nested within each other in order to representatively describe their behavior (e.g., visual or functional). The very top of the tree form hierarchy 100 may be a desktop element 101 that is representative of a computer's desktop. The desktop element 101 may have represented within it several application program elements (e.g., 105A and 105B). A typical Microsoft® Windows desktop may have several instances of applications such as Microsoft® WordPad (105A), Microsoft® Excel® and other applications (105B). At a lower level in the hierarchy 100 may be several frames (e.g., 110A, 110B, 110C, 110D, and 110E) that may be associated with application programs (e.g., 105A and 105B). For instance, a word processing application may have several frames for interacting with a user.
Each of the frames (e.g., 110A “toolbar”) may comprise of several component elements (e.g., 115A, 115B, and 115C representing list boxes, combo boxes, drop down menus, etc.). These UI elements may themselves comprise other UI elements. For example, a dialog box, or a combo box in turn may contain other control UI elements such as a button control and such nesting can go on much further depending on the UI and its component elements. Furthermore, each of the various elements in a single hierarchy may be implemented using different UI frameworks (e.g., Microsoft® Windows 32, Microsoft® HTML, etc.)
Individual UI elements or programmatic composites thereof implemented as objects may comprise interfaces through which their methods and properties are made accessible to other programs or objects thereof. FIG. 2 illustrates one method of implementing a collection of such interfaces wherein a client program 210 such as a test program or an assistive technology program uses a collection of interfaces (e.g., 215 A-B and 220 A-B) to programmatically interact with an individual UI element directly.
Typically, a client program 210 resides in one process 205 (e.g., a single virtual memory map) whereas the UI elements that are targeted (e.g., 205A and 210A) reside in another process (e.g., 225). Calls from a client process to another process for manipulating, finding, using, requesting information from and otherwise interacting with UI elements across a process boundary may be slower and consume considerably more computing resources than calls made within the process. For instance, in a multi-threading system such a cross-process call may require a greater number of system operations as a result of changing the memory map and coordinating between threads. Unfortunately, the existing Application Programming Interfaces (API) (e.g., Microsoft® Active Accessibility, MSAA hereinafter) used by accessibility aid applications require these costly cross-process calls when interacting with individual UI elements of a target UI hierarchy.